I'm a bit surprised that this declaration is required. Cython already provides a declaration for this function and it would be surprised if it didn't have the right ref-counting behavior.

(also, by the way, if we require Cython 3.1 we will gain implicit std::string_view conversion to/from bytes objects)

Importing PyBytes_FromStringAndSize via from cpython.bytes cimport PyBytes_FromStringAndSize works.

I have copied this from io.pxi, maybe it is not needed there as well / any more:

Of course this means the bytes object payload won't be cleared securely like the SecureString is.

We could try to tackle this as a followup issue/PR. bytes objects being immutable, this might be a bit delicate...

Ok, even the cryptography package doesn't do anything about it, so we may just have to live with it:
https://cryptography.io/en/latest/limitations/#secure-memory-wiping

However, cryptography does not clear memory by default, as there is no way to clear immutable structures such as bytes. As a result, cryptography, like almost all software in Python is potentially vulnerable to this attack. The CERT secure coding guidelines assesses this issue as “Severity: medium, Likelihood: unlikely, Remediation Cost: expensive to repair” and we do not consider this a high risk for most users.

Yes, I deliberately left this piece of work for a separate PR.

Happy to see this as "Likelihood: unlikely, Remediation Cost: expensive to repair" as well.

It'd be nice to find if modern Cython allows us to make use of make_shared. At least it has a declaration for it:
https://github.com/cython/cython/blob/b862800bda6cc5f540d3b7961350cd2b6ad00dbd/Cython/Includes/libcpp/memory.pxd#L106-L107

Works, this is much nicer, thanks!

I wonder why this is expecting a shared_ptr output, is it because of Cython's limited support for move-only types?

I made it work with CSecureString* out and CSecureString& out. What do you prefer?

I wonder if a cdef: block is really required, especially as this is already a cdef function. Cython nowadays is more flexible than it used to be.

Cython doesn't like the definition of a C_string:

Error compiling Cython file:
------------------------------------------------------------
...


cdef void _cb_unwrap_key(
        handler, const c_string& wrapped_key,
        const c_string& master_key_identifier, CSecureString& out) except *:
    c_string cstr
             ^
------------------------------------------------------------

pyarrow/_parquet_encryption.pyx:323:13: Syntax error in simple statement list

Without the cdef of c_string cstr, it fails with

Error compiling Cython file:
------------------------------------------------------------
...
        const c_string& master_key_identifier, CSecureString& out) except *:
    mkid_str = frombytes(master_key_identifier)
    wk_str = frombytes(wrapped_key)
    key = handler.unwrap_key(wk_str, mkid_str)
    cstr = tobytes(key)
    out = CSecureString(move(cstr))
                       ^
------------------------------------------------------------

pyarrow/_parquet_encryption.pyx:327:23: no suitable method found

You are right, cdefs can be removed by using proper casting / c-type annotation: 79cd06e